Radar and camera-based data fusion

ABSTRACT

A method including detecting an object within a field of view of a radar using a radar signal; tracking movement of the object through the field of view of the radar; triggering a camera to capture a plurality of images of the object based on the movement of the object; detecting the object in the plurality of images; combining data of the radar signal with data of the camera to estimate a position of the object; identifying a radar signal track generated by the motion of the object based on the combined data; and estimating a trajectory of the object based on identifying the radar signal track.

FIELD

The embodiments discussed herein are related to analyzing the trajectoryof moving objects.

BACKGROUND

In some cases, it may be desirable to analyze various characteristics ofa moving object. In one example, the desire to analyze the movement ofthe object may be related to participating in a sport where the movingobject may be part of the sport itself, such as a baseball, golf ball,tennis ball, hockey puck, cricket ball, ammunition for skeet or targetshooting, and the like.

Some potential problems with analyzing the trajectory of a movingobject, however, include the processing cost and difficulty of obtainingand analyzing data. The cost for the equipment, both to obtain the data,as well as to process the data, may prove cost-prohibitive forindividuals desiring a way to analyze the characteristics of the movingobjects.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above; rather, this background is only provide toillustrate one example technology area where some embodiments describedherein may be practiced.

SUMMARY

One embodiment of the present disclosure may include a method includingdetecting an object within a field of view of a radar using a radarsignal; tracking movement of the object through the field of view of theradar; triggering a camera to capture a plurality of images of theobject based on the movement of the object; detecting the object in theplurality of images; combining data of the radar signal with data of thecamera to estimate a position of the object; and estimating a trajectoryof the object based on combining the radar track with camera images.

Another embodiment of the present disclosure may include a systemincluding a processor communicatively coupled to a radar device, acamera and memory, configured to read instructions stored on the memorycausing the processor to detect an object within a field of view of aradar using a radar signal; track movement of the object through thefield of view of the radar; trigger a camera to capture a plurality ofimages of the object based on the movement of the object; detect theobject in the plurality of images; combining data of the radar signalwith data of the camera to estimate a position of the object; identify aradar signal track generated by the motion of the object based on thecombined data; and estimate a trajectory of the object based onidentifying the radar signal track.

Another embodiment of the present disclosure may include anon-transitory computer-readable medium storing computer-executablecode, the code executable by a processor to cause a system to detect anobject within a field of view of a radar using a radar signal; trackmovement of the object through the field of view of the radar; trigger acamera to capture a plurality of images of the object based on themovement of the object; detect the object in the plurality of images;combining data of the radar signal with data of the camera to estimate aposition of the object; identify a radar signal track generated by themotion of the object based on the combined data; and estimate atrajectory of the object based on identifying the radar signal track.The object and advantages of the embodiments will be realized andachieved at least by the elements, features, and combinationsparticularly pointed out in the claims. It is to be understood that boththe foregoing summary and the following detailed description areexplanatory and are not restrictive of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 illustrates an example system for radar and camera-based datafusion, in accordance with one embodiment;

FIG. 2 illustrates an elevation view of an example system for analyzinga moving object, in accordance with one embodiment;

FIG. 3 illustrates an elevation view of an example environment for radarand camera-based data fusion, in accordance with one embodiment;

FIG. 4 illustrates a flowchart of an example processing method for radarand camera-based data fusion, in accordance with one embodiment;

FIG. 5 illustrates a flowchart of an example processing method for radarand camera-based data fusion, in accordance with one embodiment;

FIG. 6 illustrates an example view of an example environment for radarand camera-based data fusion, in accordance with one embodiment;

FIG. 7 illustrates a flowchart of an example method for radar andcamera-based data fusion, in accordance with one embodiment; and

FIG. 8 illustrates an example computer system for use in radar andcamera-based data fusion, in accordance with one embodiment.

DESCRIPTION OF EMBODIMENTS

Some embodiments described herein relate to determining characteristicsof a moving object using a radar device and a camera system. Bycapturing high speed imaging data (e.g., obtained by a camera) andradar-based data simultaneously, the captured data may be fused togetherand analyzed to determine an accurate reconstruction of the trajectoryof a moving object. Using the captured data, trajectory and other motioncharacteristics of the moving object may be reconstructed and/orextrapolated. For example, the captured data obtained may include speed,velocity, rotation, axis of rotation, speed of rotation, vertical angleof elevation, azimuth angle, trajectory, release angle, etc. The motioncharacteristics may be used to further characterize the path of themoving object for later analysis. For example, if the moving object is abaseball, the flight characteristics may be used to characterize thetype of pitch (e.g., a fastball, curveball, slider, or breaking ball) orlocation of pitch (e.g., strike or ball), and a sportsman may be able toadjust his or her pitching or batting strategies based on the analysis.

In another embodiment, radar data may be unavailable or unusable (e.g.,due to the angle of the motion of the object), and thus the method mayinclude reconstructing three-dimensional trajectory data without usingthe radar data.

Rather than solving a routine problem using routine computer algorithms,some embodiments of the present disclosure may relate to usingparticular hardware components, such as radar devices and camerascontrolled by one or more processors, and techniques to address specificproblems associated with analyzing moving objects, such as baseballs,tennis balls, golf balls, cricket balls, etc. In addition, the methodsand systems described herein may be used in conjunction with artificialintelligence and/or computer learning to improve the execution of themethods described and the processing speed and efficiency of the systemsdescribed.

FIG. 1 illustrates an example system 100 for radar and camera-based datafusion, in accordance with one embodiment. The system 100 may include aradar device 110, a camera 120, a processor 130, a memory 140, acommunication device 150, and an accelerometer 160. The operation of thesystem 100 may be controlled by the processor 130, and the processor 130may be in communication with each of the other components of the system100. The components of the system 100 may work cooperatively using oneor both of radar readings from the radar device 110 and images capturedby the camera 120 to analyze characteristics of a moving object. Whileall components of the system 100 are illustrated in communication withthe processor 130 but not depicted as being in communication with eachother, any of the other components may also be in communication witheach other; for example, the radar device 110 may be in communicationwith the camera 120 and the camera 120 may be in communication with thememory 140 and the communication device 150, etc. Additionally, whilethe system 100 is illustrated as a unitary device, one or more of thecomponents may be distributed or may span across multiple devices.

The radar device 110 may include any system, component, or series ofcomponents configured to transmit one or more microwaves or otherelectromagnetic waves towards a moving object and receive a reflectionof the transmitted microwaves back, reflected off of the moving object.The radar device 110 may include a transmitter 112 and a receiver 114.The transmitter 112 may transmit a microwave through an antenna towardsthe moving object. The receiver 114 may receive the microwave reflectedback from the moving object. The radar device 110 may operate based ontechniques of Pulsed Doppler, Continuous Wave Doppler, Frequency ShiftKeying Radar, Frequency Modulated Continuous Wave Radar, or other radartechniques as known in the art. The frequency shift of the reflectedmicrowave may be measured to derive a radial velocity of the movingobject, or in other words, to measure the speed at which the movingobject is traveling towards the radar device 110. The radial velocitymay be used to estimate the speed of the moving object.

The radar device 110 may also include any of a variety of signalprocessing or conditioning components; for example, the radar device 110may include an analog front end amplifier and/or filters to increase thesignal-to-noise ratio (SNR) by amplifying and/or filtering out highfrequencies or low frequencies, depending on the moving object and thecontext in which the radar device 110 is being used. In someembodiments, the signal processing or conditioning components mayseparate out low and high frequencies and may amplify and/or filter thehigh frequencies separately and independently from the low frequencies.In some embodiments, the range of motion of the object may be a fewmeters to tens of meters, and thus, the radar bandwidth may be narrow.

The radar device 110 may initially detect the object of interest whenthe object is within the field of view of the radar or when the objectinitially enters the field of view of the radar. In some embodiments,the radar signal is tracked for a pre-determined duration of time. Atsome triggering point during the pre-determined duration of time, thecamera 120 is triggered to begin taking photographs.

The camera 120 of the system 100 may include any device, system,component, or collection of components configured to capture images.Although one camera 120 is illustrated with reference to FIG. 1, anynumber of cameras may be contemplated. The camera 120 may includeoptical elements such as, for example, lenses, filters, holograms,splitters, etc., and an image sensor upon which an image may berecorded. Such an image sensor may include any device that converts animage represented by incident light into an electronic signal. The imagesensor may include a plurality of pixel elements, which may be arrangedin a pixel array (e.g., a grid of pixel elements); for example, theimage sensor may comprise a charge-coupled device (CCD) or complementarymetal-oxide-semiconductor (CMOS) image sensor. The pixel array mayinclude a two-dimensional array with an aspect ratio of 1:1, 4:3, 5:4,3:2, 16:9, 10:7, 6:5, 9:4, 17:6, etc., or any other ratio. The imagesensor may be optically aligned with various optical elements that focuslight onto the pixel array, for example, a lens. Any number of pixelsmay be included such as, for example, eight megapixels, 15 megapixels,20 megapixels, 50 megapixels, 100 megapixels, 200 megapixels, 600megapixels, 1000 megapixels, etc.

The camera 120 may operate at certain frame-rates, or be able to capturea certain number of images in a given time. The camera 120 may operateat a frame rate of greater than or equal to about 30 frames per second.In a specific example, camera 120 may operate at a frame rate betweenabout 100 and about 300 frames per second. In some embodiments, asmaller subset of the available pixels in the pixel array may be used toallow for the camera 120 to operate at a higher frame rate; for example,if the moving object is known or estimated to be located in a certainquadrant, region, or space of the pixel array, only that quadrant,region, or space may be used in capturing the image allowing for afaster refresh rate to capture another image. Using less than the entirepixel array may allow for the use of less-expensive cameras while stillenjoying a higher effective frame rate.

Various other components may also be included in the camera 120. Suchcomponents may include one or more illuminating features such as a flashor other light source, a light diffuser, or other components forilluminating an object. In some embodiments, the illuminating featuresmay be configured to illuminate the moving object when it is proximatethe image sensor, for example, when the moving object is within threemeters of the image sensor.

Any number of a variety of triggers may be used to cause the camera 120to capture one or more images of the moving object. By way ofnon-limiting examples, the camera may be triggered when the movingobject is known or estimated to be in the field of view of the camera120, when a moving object first begins or modifies its flight (forexample when a baseball is pitched, when a baseball is batted, when agolf ball is struck, when a tennis ball is served, when a cricket ballis bowled, etc.), when a moving object is detected at a leading row ofpixels in the pixel array, etc. Another example of a trigger may be apersisting peak in a spectrum of reflected microwaves. For example, ifthere is consistently a peak at a given frequency known to be in anexpected moving object frequency for a given duration of time, this mayact as a triggering event.

In some embodiments, the camera 120 may have a field of view in whichimages may be captured. The field of view may correspond to the pixelarray. In some embodiments, the field of view may be limited such thatthe moving object only spends a limited amount of time within the fieldof view. In such embodiments, the camera 120 may be triggered to captureimages while the moving object is within the field of view. The time inwhich the moving object is within the field of view of the camera 120may be referred to as an optimal photograph timeframe. In someembodiments, the optimal photograph timeframe may include when only theentire moving object is within the field of view or may include whenonly a portion of the moving object is within the field of view. Otherfactors may also contribute to the optimal photograph timeframe, such asthe distance between the image sensor and the moving object, the amountof illumination that may be provided by an illuminating feature, etc.For example, the optimal photograph timeframe may occur when the movingobject is traveling between three meters and one meter away from thecamera 120 as that may be where a flash of the camera 120 providesillumination of the moving object.

In some embodiments, the field of view of the camera and the radar maybe the same or may be different. In the case where the fields of view ofthe camera and the radar are different, a trigger mechanism may operateto ensure that the moving object remains in the field of view of thecamera for as long as photographs are being captured.

The processor 130 may include any suitable special-purpose orgeneral-purpose computer, computing entity, or processing deviceincluding various computer hardware or software modules and may beconfigured to execute instructions stored on any applicablecomputer-readable storage media; for example, the processor 130 mayinclude a microprocessor, a microcontroller, a digital signal processor(DSP), an application-specific integrated circuit (ASIC), aField-Programmable Gate Array (FPGA), or any other digital or analogcircuitry configured to interpret and/or to execute program instructionsand/or to process data. Although illustrated as a single processor inFIG. 1, the processor 130 may include any number of processorsconfigured to perform, individually or collectively, any number ofoperations described in the present disclosure. Additionally, one ormore of the processors may be present on one or more differentelectronic devices, such as different devices physically coupledtogether or communicating remotely. By way of example, one or more ofthe processors being in different devices may include a processingsub-component associated with one or more of the radar device 110, thecamera 120, the communication device 150, the accelerometer, or anyother device coupled to or associated with the system 100, working witha processor in another device such as a remote display device or aremote computing device configured to perform one or more of theoperations described in the present disclosure.

In some embodiments, the processor 130 may interpret and/or executeprogram instructions and/or process data stored in the memory 140. Insome embodiments, the processor 130 may fetch program instructions froma data storage and load the program instructions in the memory 140.After the program instructions are loaded into memory 140, the processor130 may execute the program instructions. In some embodiments, theexecution of instructions by the processor 130 may direct and/or controlthe operation of the device 100. For example, the processor 130 mayinstruct the camera 120 to capture images at certain times, for example,during the optimal photograph timeframe.

In some embodiments, the processor 130 may use the received microwavesignal from the radar device 110 to determine the radial velocity of themoving object. The processor 130 may perform spectral measurement on theDoppler signal to determine whether there is a moving object; forexample, using the example of a pitched baseball, the spectralmeasurements may be used to determine if there is an object reflectingthe microwaves travelling between forty and one hundred miles per hour.The spectral measurements may include performing a Fast FourierTransform (FFT) to determine whether any of the frequency peaks cross aparticular threshold. The processor 130 may then use that peak todetermine the radial velocity of the moving object.

The processor 130 may use the determined radial velocity as one factorto trigger the camera 120 to capture images of the moving object. In oneexample, a flight path distance and/or trajectory may be estimatedbetween an initial point of the moving object and the system 100. Insome embodiments, the flight path distance may be estimated as thedistance between the initial point of the moving object and the field ofview of the camera 120; for example, if the moving object is a baseball,the estimated flight path distance may be the distance between thepitcher's mound and home plate. The processor 130 may then determine theamount of time required for the moving object to traverse the flightpath distance and arrive at the field of view of the camera 120 or untilthe optimal photograph timeframe. The processor 130 may also account forany time required for the camera 120 to actually capture the image.

In some embodiments, the processor 130 may determine the number ofimages to be captured. The number of images to be captured may be basedon an input from a user, a default setting, the context of the movingobject, or selected based on the field of view of the camera 120 and thespeed of the moving object. A larger number of images may allow for amore accurate extrapolation of flight characteristics, such as thetrajectory. In some embodiments, an example default setting may includecapturing two, three, or four images. The processor 130 may beconfigured to trigger the camera 120 to capture images of the movingobject during a pre-determined photograph timeframe such that thedesired number of images may be captured during the pre-determinedphotograph timeframe; for example, the processor 130 may determine thattwo images are desired and may wait until the entire moving object is inthe field of view and may then trigger the camera 120 to capture twosuccessive images as quickly as possible with the camera 120 with thefirst image only using one quadrant of the image sensor of the camera120 and the second image only using a different quadrant of the imagesensor of the camera 120.

The processor 130 may be configured to perform processing of thecaptured images or may use the communication device 150 to transmit theimages to another device to perform processing of the captured images.Processing of the captured images may include extrapolating flightcharacteristics of the moving object; for example, the processor 130 maydetermine speed, velocity, rotation, axis of rotation, speed ofrotation, vertical angle of elevation, azimuth angle, trajectory,release angle, etc.

In some embodiments, the processor 130 may perform othercharacterizations of the moving object based on one or more of theflight characteristics. Using a pitched baseball as an example, theprocessor 130 may use the trajectory to determine whether the pitch is aball or a strike. As another example, the pitch may be characterized asa fastball, breaking ball, slider, etc. based on the speed, rotation,trajectory, etc. of the baseball.

The processor 130 may compare the extrapolated flight characteristics tothe data collected from the radar device 110; for example, anextrapolated trajectory may be compared to the actual radar valuesmeasured at one or more points in time. If there is a discrepancybetween the radar values and the extrapolated trajectory, the processor130 may include one or more environmental factors or atmosphericconditions to account for the discrepancy, such as humidity, wind, rain,etc. Additionally or alternatively, the processor 130 may verify theimage quality of the captured images used to extrapolate the flightcharacteristics; for example, if any of the captured images areparticularly fuzzy or grainy, the processor 130 may modify theextrapolated flight characteristics to conform more closely to themeasured values from the radar device 110. Other operations of theprocessor 130 with respect to the system 100 may be described withfurther reference to other figures in the present disclosure, describedbelow.

The memory 140 may include computer-readable storage media for carryingor having computer-executable instructions or data structures storedthereon, including one or more of the images captured by the camera 120.Such computer-readable storage media may include any available mediathat may be accessed by a general-purpose or special-purpose computer,such as the processor 130. By way of example, and not limitation, suchcomputer-readable storage media may include tangible or non-transitorycomputer-readable storage media including random access memory (RAM),read-only memory (ROM), electrically erasable programmable read-onlymemory (EEPROM), compact disk-read-only memory (CD-ROM) or other opticaldisk storage, magnetic disk storage or other magnetic storage devices,flash memory devices (e.g., solid state memory devices), or any otherstorage medium which may be used to carry or store desired program codein the form of computer-executable instructions or data structures andwhich may be accessed by a general-purpose or special-purpose computer.Combinations of the above may also be included within the scope ofcomputer-readable storage media. Computer-executable instructions mayinclude, for example, instructions and data configured to cause theprocessor 130 to perform a certain operation or group of operations. Insome embodiments, memory 140, while depicted as a single component, maybe multiple components. For example, memory 140 may be implemented as acombination of RAM, ROM, and flash memory.

The communication device 150 may be any component, device, orcombination thereof configured to transmit or receive data. Thecommunication device 150 may include, without limitation, a modem, anetwork card (wireless or wired), an infrared communication device, awireless communication device (such as an antenna), and/or chipset (suchas a Bluetooth device, an 802.6 device (e.g., Metropolitan Area Network(MAN)), a WiFi device, a WiMax device, cellular communicationfacilities, etc.), and/or the like. The communication device 120 maypermit data to be exchanged with a network (such as, either alone or inany suitable combination, the Internet, an Intranet, a local Wi-Finetwork, a wireless Local Area Network (LAN), a mobile network (e.g., a3G, 4G, 5G, and/or LTE network), a LAN, a Wide Area Network (WAN), aMAN, a Bluetooth connection, or any other suitable communicationnetwork) and/or any other devices described herein, including remotedevices.

The accelerometer 160 may measure an angle of the system 100 for use bythe radar device 110 and/or the camera 120 to provide correct planarinformation for planar equalization. The accelerometer, for example, maybe used to calibrate the camera and to ensure that the camera may tiltand roll with a pre-determined accuracy. In some embodiments, the use ofthe accelerometer may be used to calibrate the camera in such a way toenable the conversion of camera measurements to world coordinates (e.g.,spherical or Cartesian coordinates). In addition, the accelerometer maybe used to align the camera data with the radar data.

Modifications, additions, or omissions may be made to the system 100without departing from the scope of the present disclosure; for example,in some embodiments, the system 100 may include any number of othercomponents that may not be explicitly illustrated or described; forexample, the system 100 may not include the communication device 150 andmay perform all image processing locally at the system 100. The system100 may also include a display for displaying any of the flightcharacteristics or other characterizations performed by the system 100.

FIG. 2 illustrates an elevation view of an example environment 200 forradar and camera-based data fusion, in accordance with one embodiment.For purposes of clarity and brevity, the remaining portion of thedetailed disclosure uses as an example of the moving object a baseball206, where the example environment is a baseball diamond. In FIG. 2, theplacement of the system device 100 may be positioned at home plate, witha pitcher standing at location 204. In FIG. 2, the system 100 may berepresentative of system 100 described with respect to FIG. 1. It shouldbe understood, however, that any moving object may be contemplated andthat present disclosure is applicable to any other moving object, forexample, other sporting paraphernalia such as a tennis ball, a golfball, a hockey puck, a cricket ball, etc.

The system 100 may be located in any of a variety of locations such thatthe radar device 110 and the camera may have the moving object in thefield of view 210 for a length of time sufficient to capture a pluralityof images with the camera. In one embodiment, the system 100 is locateda known distance 214 from a reference point (such as pitching mound204). In some embodiments, the system 100 may be located on or near theground at home plate 202 and angled upwards degrees (as illustrated byangle 212). In some embodiments, the system 100 may be elevated acertain height (not illustrated in FIG. 2) behind the home plate 202 andangled downward. In other embodiments, the system 100 may be to one sideof the home plate 202 and angled towards the pitching mound 204. Thesystem 100 may be located in any of a variety of locations such that theradar device 110 and the one or more cameras may have the moving objectin the field of view 210 for a length of time sufficient to capture aplurality of images with the camera.

In one example embodiment, the baseball 206 may be pitched from apitching mound 204, and be moving along a path 208 starting at time toand where the baseball may be may be at time t_(N-1) (now illustrated bydotted line 206-b). Baseball 206 may be considered an “incoming object”with respect to system 100 in this example. In another example, thebaseball 206-c may be hit by the batter located at home plate 202 movingalong a path 218 illustrated between baseball 206-c at time to and attime t_(N-1) (now illustrated by dotted line 206-d). Baseball 206-c maybe considered an “outgoing object” with respect to system 100 in thisexample.

FIG. 3 illustrates an elevation view of an example environment for radarand camera-based data fusion, in accordance with one embodiment.Baseball 206 may be pitched from pitching mound 204 towards home plateand also the system 100. The system 100 may be positioned at an angle212 to increase a field of view 206 of a camera of the system 100. Insome embodiments, the radar device and the camera of the system 100 maybe tilted to the same plane. Angling the radar device and the camera tothe same plane may facilitate easier manufacturing. In some embodiments,the radar device and the camera of the system 100 may be tilted todifferent planes.

When the baseball 206 is pitched, the system 100 may transmit microwavestowards the baseball 100. In some embodiments, the system 100 mayalready be transmitting microwaves 216 in all directions with spatialdistribution described by an antenna beam pattern of a radar device ofthe system 100. The transmitted microwave 216 may be reflected off ofthe baseball 206 and because of the relative velocity of the baseball206, the reflected microwave 218 may have a different frequency than thetransmitted microwave 216. The frequency drift may be referred to asDoppler frequency and may be a factor of the radial velocity 220.

The radar device 110 of the system 100 may facilitate the determinationof the radial velocity 220 of the baseball 206; for example, a vectordiagram 222 may show a radial velocity 220 of the baseball 206. Thebaseball 206 may have component vectors 224 and 226 contributing to theradial velocity 220. In some embodiments, if the distance between themoving object (e.g., the baseball 206) is much larger than the distanceabove the ground of the moving object (e.g., the height of the baseball206 above the ground), the radial velocity 220 may be estimated to bethe same as the horizontal velocity 224 of the moving object. Statedanother way, if the vertical velocity 226 of the radial velocity 220 ismuch smaller than the horizontal velocity 224, the radial velocity 220may be estimated to be the same as the horizontal velocity 224. Oneexample of the horizontal velocity 224 being much larger than thevertical velocity 226 may be if the horizontal velocity 224 is an orderof magnitude larger than the vertical velocity 226.

Tracking a Regularly Shaped Object

FIG. 4 illustrates a flowchart of an example processing methodology forradar and camera-based data fusion, in accordance with one embodiment.At block 402, an object may be detected within a field of view of radardevice 100, which may trigger the camera to being taking photos of theobject. When pre-determined tracking conditions are satisfied, thecamera is triggered to begin taking photographs. In one example, thetrigger to begin taking photographs is that a pitched ball appears fieldof view of the radar. Simultaneously, the motion of the hand pitchingthe ball may be detected, and thus, the motion of the ball may betracked both as an incoming object and an outgoing object. In anadditional or alternative example, the trigger to begin takingphotographs may be identifying the motion of a ball when hit fromoff-a-tee. In this example, the radar signal from swinging of the handis detected and as a result triggers the camera.

In an addition or alternative example, the trigger to begin takingphotographs may be detection of the hand swing. For example, the system100 may be placed on a tripod in such a way that the hands of a userswinging, e.g., a bat or a golf club, are visible. The swing of thehands can be tracked and the camera can be triggered to begin takingphotographs. In an additional or alternative example, the swing of thehands may be tracked, and the swing data may be correlated with apre-defined mask until a threshold parameter is met (e.g., amplitude,speed, etc.). The correlation signal may also be a time-domain referencesignal.

In some embodiments, when the camera is triggered to begin takingphotographs, the system may take N photographs, where N may bepre-determined by a user, calculated by the system based on previousrounds of photography, determined by a manufacture, and/or determined byany number of conditions which may alter the number of photos taken.

Based on the trigger mechanism, the N images are taken at time t₀, t₁, .. . t_(N-1), with a sampling interval of dt, such thatt _(n) =t _(n-1) +dt  (1)where n ∈ {0, . . . , N−1}.

It is not necessary that the images be taken at a periodic interval ofdt, and the images may be taken at any time interval such as anaperiodical time interval. Regardless of the time interval, the totaltime duration of the images taken, DT=t_(N-1)−t₀, should be selected insuch a way that the fastest expected object remains in the field of viewof the camera.

The lower bound on dt may be determined by practical imitations, such asthe camera shutter speed and/or hardware limitations. The camera,however, should be a high speed camera with a frame rate greater than 60frames per second in order to capture enough images of the moving objectwhile the moving object is still in the field of view of the camera.Image processing algorithms, as described here, may be applied to theseimages to extract the position information of the moving object at eachtime t₁. The position of the moving object at each time sample may bestored in a matrix P^(C) containing vectors P_(n) ^(C) as the columns,as in the following manner:P ^(C):=[P ₀ ^(C) . . . P _(N) ^(C) . . . P _(N-1N) ^(C)],n∈{0, . . .,N−1}  (2)where P₀ ^(C):=[x_(n) y_(n) z_(n)]^(T).

After the trigger mechanism activates, initiating the photography, atblock 404 the system operates to detect the object (e.g., the baseballor generally a moving object) in the photographs taken during block 402.In some cases, if the system operates using a single camera, it may beuseful to have previously knowledge about the shape and size of theobject; for example, that the object is a baseball as opposed to a golfball or a football, having an approximate diameter of 2.5 inches. Insome cases, if the system operates using multiple cameras (e.g., astereo camera system), knowing the shape and size of the object ofinterest beforehand may not be needed, but may speed up processing ifthe data can be provided to the system ahead of time.

In some embodiments, the system 100 has prior knowledge, based on themotion-based trigger mechanism, that the object is or will be in motionwith respect to a static background (e.g., the motion of a hit ballacross a static baseball field environment). Thus, in one example,detecting the object of interest may be done by using the firstphotograph of a series of captured photographs as a background image,and then subtracting images from each subsequent photograph from theinitial background photograph. The subtraction operation may be followedby a thresholding and/or applying a filter to remove noise and thusdetect the object against the background.

In another embodiment, detecting the object in the photographs may beenabled by first selecting a photograph from the series of photograms asthe “starting” image. Subsequently, images from the photograph aresubtracted from photographs occurring before the “starting” photographand photographs occurring after the “starting” photograph. The differentphotographs are multiplied to highlight the portions of the photographsthat are common to both the before and after images. The result of thismultiplication further highlights the region of interest inside eachphotograph where the moving object can be found. If the object haswell-defined characteristics (e.g., a circular shape, an oval shape,etc.), then the system 100 can use pattern-matching using pre-knownpatterns, to determine the object in the images.

In another embodiment, detecting the object may further include using aHough transform for objects that may be parameterized using knownparameters; for example, a circle has three parameters (i.e., radius,and horizontal and vertical position of the circle's center); an ellipsehas four parameters (i.e., the major axis, the minor axis, andhorizontal and vertical positions of the ellipse's center). Once theobject is detected as being present in the images, the parameters ofinterest may be stored in an array, with each entry having a time-stamp(e.g., using an internal clock of the system 100 or another timingdevice) that may aid in tracking the path of the object.

Once the object has been detected, at block 406, the system 100estimates the position of the object in the three-dimensional space. Theposition of the object in three-dimensional space, for each photographtaken, is associated with a time t_(i) for every i^(th) photograph.Given the accurate position of the object at each time sample, and usinginterpolation, an accurate trajectory may be reconstructed for theduration of the observation time period (i.e., a pre-determinedtimeframe).

In some cases, the horizontal and vertical position of the object may beaccurately determined, however, the camera observation for depth may notbe as accurate. In terms of spherical coordinates, the range has agreater error as compared to the azimuth and elevation. The error inranging may thus have an effect on the error in the position on allthree dimensions which may lead to an error in finding the speed anddirection in which the object is moving.

At block 408, the object may be identified based on the camera-basedmeasurement that is closest to the system 100. In some cases, the objectclosest to the system 100 may be used because the error in ranging maybe less than an object located farther from the system 100; the fartherthe object is located from the system, the range resolution may beexponentially worse.

At block 410, the system 100 may identify a radar track generated by themoving object using the camera-based measurements determined in theblocks above. The radar device 110 may generate a Doppler signal thatcorresponds to the radial velocity of the moving object, where theradial velocity may be used to correct for the trajectory.

In some embodiments, radar device 110 may capture data comprising twoanalog components: the in-phase component (or the I-Channel) and thequadrature component (or the Q-channel). When taken together, thein-phase component and the quadrature component form a complex signals(t), where:s(t)=I(t)+iQ(t),  (3)where i is equal to √{square root over (−1)}. The in-phase component andthe quadrature component are sampled using an analog-to-digitalconverter at a sampling frequency F_(s). The components may bepre-filtered and amplified as needed before sampling. After sampling ahigh order Finite Impulse Response (FIR) digital filter is applied toeach channel. In some embodiments, an Infinite Impulse Response Filter(IIR) may be applied to the sample instead of the FIR filter. In somecases, the filter removes low frequency motion generated by, forexample, the motion of an individual (e.g., the pitcher or the batter,in this example), limb motion other than the motion of the limb ofinterest, and the like. At this point, the data may be in thetime-domain, and using moving window N-point Fast Fourier Transform(FFT), the time-domain data is converted to time-frequency domain data.To generate a smooth spectrum with few artifacts of finite-durationwindowing and reduced spectral leakage windowing functions such asHamming, Blackman, Kaiser, and the like, may be applied to pre-multiplythe time-domain data before taking FFT.

The raw data captured by the system may be captured in the cameracoordinates system, but should be converted to the world coordinatessystem, where the world coordinates system may be spherical coordinatesor Cartesian coordinates, for example. In order to convert the data tothe world coordinate system, the camera position and orientation areused in order to construct a camera-to-world coordinates transformationmatrix: R_(C) ^(W). The camera-to-world coordinate matrix may be a 4×4matrix containing associated rotations and translations used to convertany vector from the camera into the selected world coordinate system. Avector in world coordinates may be obtained by the following equation:S ^(W) =R _(C) ^(w) S ^(C)  (4)where S^(W) is the transformed vector into world coordinates, andS^(C):=[S_(x)S_(y)S_(z)1] is a vector in camera coordinates. The vectoris three-dimensional, however “1” may be appended as a 4^(th) dimensionto cater for translations, as R_(C) ^(W) is 4×4.

For a regularly shaped object (e.g., baseball, golf ball), having aradius, r_(b), the spherical coordinates in the camera frame ofreference, ρ_(n), θ_(n,and ϕ) _(n) , are given by

$\begin{matrix}{\rho_{n} = \frac{r_{b}}{\theta_{b}}} & (5) \\{\theta_{n} = \frac{x_{n}^{i}}{\Theta_{F}}} & (6) \\{\phi_{n} = \frac{y_{n}^{i}}{\Theta_{F}}} & (7)\end{matrix}$where θ_(b) is the angle subtended by the object, given by

${\theta_{b} = \frac{2r\;\Theta_{F}}{L}},$where r is the radius of the object in pixel, L is the total length ofthe image, Θ_(F) is the field of view of the lens, x_(n) ^(i) and y_(n)^(i) represent the raw image x and y pixel values of the center locationof the n^(th) object.

At step 412, the system 100 may estimate the trajectory of the movingobject and calculate the parameters of interest from the estimatedtrajectory, where the parameters of interest may include speed,velocity, rotation, axis of rotation, speed of rotation, vertical angleof elevation, azimuth angle, trajectory, release angle, etc.

Multiple Object Tracking

FIG. 5 illustrates a flowchart of an example processing method 500 forradar and camera-based data fusion, in accordance with one embodiment.In some embodiments, more than one object may be tracked, eitherpurposefully or because there may be multiple moving objects that theradar may detect (e.g., other balls, birds, planes, people in the fieldof view). In the case of multiple objects it may be difficult to trackthe correct objects; therefore, in one example, the radar used may be anarrow beam radar having a predefined beam-width. If the moving objectis in the beam of the radar, the object will generate a Dopplerfrequency equivalent to the radial velocity. The objects trackedsimultaneously may include, but are not limited to, the hand of thepitcher, the ball, the arm of the pitcher and/or the batter, the swingof a bat or golf club, and the like.

The detection may be based on calculating a signal-to-noise ratio (SNR),as shown in block 502. In block 504, the identified frequency orfrequencies may be associated with existing pre-determined radar tracksstored in a radar track pool based on proximity. A determination ifthere is an association may be made in block 506. If not, as shown inblock 508, then a new radar track is created (block 508) and placed inthe radar track pool (block 510).

If an association with an existing track is determined to be so (block506), then the pre-associated radar track is determined to be present inthe radar track pool (block 510). At each iteration, the radar trackdata may be used to predict the next expected frequency detected (block516). If the detection for a certain radar track fails for multipleiterations (e.g., the detection of one object among multiple objects, ordistinguishing multiple objects from one another fails), then the radartrack is starved (block 512), and subsequently the radar track isdeleted from the radar track pool (block 514). On the other hand, if theradar track does not fail (e.g., an object is detected from a group ofmultiple objects, or multiple objects are distinguished from eachother), then the radar track is updated (block 518) and entered into theradar track pool (510) for a later association.

In one embodiment, the following state-space tracker equation may beused for multiple object tracking:x _(n) =Px _(n-1) +G(y−HPx _(n-1))  (8)where vector x is the state vector.

Vector x may be represented by:

$\begin{matrix}{x = \begin{matrix}x \\\overset{.}{x} \\\overset{¨}{x}\end{matrix}} & (9)\end{matrix}$

The matrix G may be the gain matrix. A Kalman filter may be used to findG, or in an alternative, static values of α, β, and γ may be used tofind G as follows:

$\begin{matrix}{\begin{matrix}\alpha & 0 & 0 \\0 & \beta & {0\quad} \\0 & 0 & \gamma\end{matrix}\quad} & (10)\end{matrix}$

The matrix P may be the state-transition matrix as follows:

$\begin{matrix}{\begin{matrix}1 & {dt} & {dt}^{2} \\0 & 1 & {dt} \\0 & 0 & 1\end{matrix}\quad} & (11)\end{matrix}$where dt is the radar data sampling duration.

The matrix H may be the measurement matrix as follows:

$\begin{matrix}{\begin{matrix}1 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{matrix}\quad} & (12)\end{matrix}$

The radar track data may be fitted to equation (13) using a LeastSquares fitting, as follows:v(t)=β₀+β₁ t+β ₂ t ²+β₃ t ³  (13)

If the initial range is set as ρt₁, then the range ρ_(t1) at time t₁ maybe found using the following equation:

$\begin{matrix}{{\rho t_{1}} = {{\rho t_{0}} + {\frac{1}{12}\left\lfloor {{{- 12}{\beta_{0}\left( {t_{0} - t_{1}} \right)}} - {6\beta_{1}t_{0}^{2}} + {6\beta_{1}t_{1}^{2}} - {4\beta_{2}t_{0}^{3}} + {4\beta_{2}t_{1}^{3}} - {3\beta_{3}t_{0}^{4}} + {3\beta_{3}t_{1}^{4}}} \right\rfloor}}} & (14)\end{matrix}$

Given the closest range of the object ρt₀, equation (14) integrates aDoppler curve in order to find a range difference between t₁ and t₀. Inone embodiment, it may be assumed that the maximum ranging error due topixel resolution at range ρt₀ is dρ. Thus, the error at range ρt₁ may bedρ+dν(t₁−t₀). Due to cubic curve fitting, the error dν may be very smalland the contributing factor for the error remains dρ; the ranging errormay therefore remain constant for the complete duration of the baseballflight as it is tracked by the radar.

FIG. 6 illustrates an example view 600 of an example environment forradar and camera-based data fusion, in accordance with one embodiment.In one embodiment, the object may not be a sphere or an ellipsoid (orother predictably regular shape), but may instead be irregularly shaped,such as a golf club, which may present additional challenges since theshape and related shape-based parameters may not be pre-known.

The system 602 may include two cameras 604 and 606, as well as a radarunit 608. Cameras 604 and 606, and radar 608, may have fields of view612, 614, and 616, respectively. In some embodiments, a point ofinterest P 618 may be, for example, the center of the club head, or someother protrusion that may be most likely to reflect more of the radarsignal. The cameras 604 and 606 may be triggered to take images of theobject 610 (e.g., a golf club head) in order to determine a point ofinterest P 618 located on the object 610 as described above; namely,when the point of interest P 618 enters the field of view 616 of theradar, the cameras 604 and 606 may be triggered to begin takingphotographs. The three-dimensional coordinates of the point of interestP are determined. In the case where there are multiple radar reflectionsfrom different locations, such as the toe or heel of the club, anaverage value may be used.

When the coordinates of P 618 are determined, P 618 is then associatedwith the radar signal sent and received from radar unit 608. Point ofinterest P 618 then tracked over multiple photographs taken with stereocameras 604 and 608 leading up until actual contact with a golf ball (ora baseball bat hitting a baseball, for example). For example, thecoordinates of P 618 may be measured at time to and then again at timet_(N-1), as the golf club is swung. In an alternative or additionalembodiment, the coordinates of P 618 may be measured multiple timesbetween time to and then again at time t_(N-1). In one example, timet_(N-1) may be the time the golf club 610 makes contact with a golfball. At that point, the trajectory of the golf ball may be estimatedusing the methods described in this description, such as those methodsdescribed in conjunction with FIGS. 1-4.

In some cases, the range resolution may worsen as the object movesfurther away from cameras 704 and 706. In order to reduce the worseningof the range resolution, the time when the object is determined to beclosest to the camera is selected as the starting time ρt₀, with regardto equation (14). Equation (14) may then be used to obtain the range ofthe object for various other time samples, and the three-dimensionaltrajectory may be reconstructed with a similar level of accuracy asreconstructing the trajectory of a regularly shaped object as describedpreviously.

Camera-Only Trajectory Estimation

In some embodiments, the object may be moving in a direction that isperpendicular or nearly perpendicular to the radar. In such as cause,the radial speed measured by the radar may be close to zero or otherwisevery small; however, it is still possible to measure the radial speed ofthe object unless the Doppler speed is absolutely zero.

In practice, the Doppler speed is measured through detecting frequencycorresponding to the radial speed of the object. There is motion aroundthe object (e.g., motion of a pitcher, motion of a batter, motion of abat, motion of other players, etc.), that may generate interference atlow frequencies. It may be difficult to detect a moving object thatgenerates a low Doppler frequency, thus, image processing may be usedfor estimating a trajectory of the moving object.

For a moving object having some speed, the initial trajectory may beassumed to have a mostly linear motion and be moving at a constantspeed. The Doppler speed, however, depends on the position of theprojectile and the direction of the motion. In one embodiment, theclosest measurement of the moving object may be captured at time t₀. Therange ρ, azimuth θ, and elevation ϕ of the moving object may be foundusing equations (5), (6), and (7) discussed above with respect to FIG.4.

The spherical coordinates (i.e., equations (5), (6), and (7)) may betransformed into Cartesian coordinates using the following equations:x _(n)=ρ_(k) cos ϕ_(n) cos ϕ_(n)  (15)y _(n)=ρ_(k) sin ϕ_(n)  (16)x _(n)=ρ_(k) cos ϕ_(n) sin θ_(n)  (17)

The range ρ₀ may be calculated from imaging using equations (5), (6),and (7). The remaining ranges (i.e., ρ₁, . . . , ρ_(n)) are calculatedusing equation (14) described previously. The azimuth θ and elevation ϕmay be calculated by finding the center of the moving object. Thecoefficients β₀, . . . , β₂, are unknown without the benefit of radardata (i.e., unknown because the object may be perpendicular or nearperpendicular to the radar measurements). Thus, the coefficients may beestimated using the following optimization equation, where min β:Σ_(i=1) ^(N)[∥v _(i) −v _(i-1)∥₂ ²(v _(x) ₁ −v _(x) _(t-1) )²+(v _(y) ₁−v _(y) _(t-1) )²+(v _(z) ₁ −v _(z) _(t-1) )²]  (18)and where:β₀ _(min) ≤β₀≤β₀ _(max)β₁ _(min) ≤β₁≤β₁ _(max)β₂ _(min) ≤β₂≤β₂ _(max)

In one embodiment, solving equation (18) may be accomplished using agrid search for parameters β₀, β₁, and β₂. In another embodiment, agradient descent algorithm may be used to determine the parameters.

Gradient Descent

In using a gradient descent to determine the parameters, the methodfinds the Jacobian matrix of the error function of equation (18) and thedescend in the direction of the Jacobian until the optimum solution, β*,is found. The error function is given by equation (18) (above) andreproduced below:f(β)=Σ_(i=1) ^(N)[∥v _(i) −v _(i-1)∥₂ ²+(v _(x) ₁ −v _(x) _(t-1) )²+(v_(y) ₁ −v _(y) _(t-1) )²+(v _(z) ₁ −v _(z) _(t-1) )²]  (18)where f is dependent on β:=[β₀ β₁ β₂ β₃].

The Jacobian matrix is given by:

$\begin{matrix}{J = \left\lbrack {\frac{\partial f}{\partial\beta_{0}}\mspace{14mu}\ldots\mspace{14mu}\frac{\partial f}{\partial\beta_{3}}} \right\rbrack} & (19)\end{matrix}$

A gradient descent algorithm is provided below:

given an initial point β ∈ dom f (20) while iterate till stoppagecriterion met do  1) find the descent direction using Jacobian J  2)choose step size ξ using exact or backtracking line search  3) update β:= β + −J ξ end while

The stopping criterion is ∥J∥₂<η, where η is a small positive constant.The exact details of backtracking or exact line search algorithms can befound in other art. Once the solution β* is found, equation 15 may beused to find the range of the object for all times t_(i). The result maybe described in spherical coordinates, and thus may be converted toCartesian coordinates in the camera system, and in an additional oralternative embodiment, then to world coordinates.

FIG. 6 illustrates a flowchart of an example method 700, in accordancewith one embodiment. The method 700 may be performed by any suitablesystem, apparatus, or device; for example, the system 100 of FIGS. 1, 2and 3 may perform one or more of the operations associated with themethod 700. Although illustrated with discrete blocks, the steps andoperations associated with one or more of the blocks of the method 700may be divided into additional blocks, combined into fewer blocks, oreliminated, depending on the desired implementation.

At block 702, the method may include detecting an object within a fieldof view of a radar using a radar signal. For example, radar device 110may detect the presence of an object using a reflected Doppler radarsignal as the object (e.g., baseball 206) enters the field of view.

At block 704, the method may include tracking a motion of the objectthrough the field of view of the radar. For example, the motion of thebaseball 206 may be tracked using radar signaling over a period of timeas the baseball 206 traverses through a distance 214.

At block 706, the method may include triggering a camera to capture aplurality of images of the object based on tracking the motion of theobject. For example, system 100 may determine that a pre-determinedtriggering threshold has been satisfied, such as detecting the entranceof the baseball 206 into the radar's field of view. As a result, camera120 may be triggered to begin taking a series of photographs as thebaseball continues to move throughout the radar's (and now also thecamera's) field of view.

At block 708, the method may include detecting the object in theplurality of images. For example, the camera 120 may take a series ofphotographs over a pre-determined period of time, or until apre-determined threshold has been met. The photographs may be analyzedby system 100 to determine the location of the baseball 206 throughoutthe series of photographs.

At block 710, the method may include combining data of the radar signalwith data of the camera to estimate a position of the object. Forexample, the camera data and the radar data may be fused to determinethe coordinates of the ball as well as parameters associated with thetrajectory of the baseball's path, in order to estimate a position ofthe baseball at each estimated time.

At block 712, the method may include identifying a radar signal trackgenerated by the motion of the object based on the combined data. Atblock 714, the method may include estimating a trajectory of the objectbased on identifying the radar signal track. For example, based on thefused data, and the calculated and estimated location and parameters,the trajectory of the baseball 206 may be estimated and stored for lateranalysis. In the case of later analysis, the calculations may be used tofurther adjust and refine later trajectory estimations.

Accordingly, the method 700 may be used to analyze a moving object.Modifications, additions, or omissions may be made to the method 700without departing from the scope of the present disclosure. For example,the operations of method 700 may be implemented in different order.Additionally or alternatively, two or more operations may be performedat the same time. Furthermore, the outlined operations and actions areonly provided as examples, and some of the operations and actions may beoptional, combined into fewer operations and actions, or expanded intoadditional operations and actions without detracting from the essence ofthe disclosed embodiments. All of the examples provided above arenon-limiting and merely serve to illustrate the flexibility and breadthof the present disclosure.

FIG. 8 illustrates an example computer system that may be employed inultrasonic communications. In some embodiments, the computer system 800may be part of any of the systems or devices described in thisdisclosure. For example, the computer system 800 may be part of any ofthe system 100 of FIGS. 1-3 and/or system 600 of FIG. 6.

The computer system 800 may include a processor 802 (which may includeprocessor 130 of FIG. 1), a memory 804 (which may include memory 140 ofFIG. 1), a file system 806, a communication unit 808, an operatingsystem 810, a user interface 812, and a module 814, which all may becommunicatively coupled. In some embodiments, the computer system mayinclude, for example, a desktop computer, a client computer, a servercomputer, a mobile phone, a laptop computer, a smartphone, a smartwatch,a tablet computer, a portable music player, a networking device, or anyother computer system.

Generally, the processor 802 may include any computer, computing entity,or processing device including various computer hardware or softwaremodules and may be configured to execute instructions stored on anyapplicable computer-readable storage media. For example, the processor802 may include a microprocessor, a microcontroller, a digital signalprocessor (DSP), an application-specific integrated circuit (ASIC), aField-Programmable Gate Array (FPGA), or any other digital or analogcircuitry configured to interpret and/or to execute program instructionsand/or to process data, or any combination thereof. In some embodiments,the processor 802 may interpret and/or execute program instructionsand/or process data stored in the memory 804 and/or the file system 806.In some embodiments, the processor 802 may fetch program instructionsfrom the file system 806 and load the program instructions into thememory 804. After the program instructions are loaded into the memory804, the processor 802 may execute the program instructions. In someembodiments, the instructions may include the processor 802 performingone or more of the actions of the methods 400 and/or 700 described withreference to FIGS. 4 and 7, respectively.

The memory 804 and the file system 806 may include computer-readablestorage media for carrying or having stored thereon computer-executableinstructions or data structures. Such computer-readable storage mediamay be any available non-transitory media that may be accessed by ageneral-purpose or special-purpose computer, such as the processor 802.By way of example, and not limitation, such computer-readable storagemedia may include non-transitory computer-readable storage mediaincluding Read-Only Memory (ROM), Electrically Erasable ProgrammableRead-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) orother optical disk storage, magnetic disk storage or other magneticstorage devices, flash memory devices (e.g., solid state memorydevices), or any other storage media which may be used to carry or storedesired program code in the form of computer-executable instructions ordata structures and which may be accessed by a general-purpose orspecial-purpose computer. Combinations of the above may also be includedwithin the scope of computer-readable storage media. Computer-executableinstructions may include, for example, instructions and data configuredto cause the processor 802 to perform a certain operation or group ofoperations, such as one or more of the actions of the methods 400 and/or700 described with reference to FIGS. 4 and 7, respectively.

The communication unit 808 may include any component, device, system, orcombination thereof configured to transmit or receive information over anetwork. In some embodiments, the communication unit 808 may communicatewith other devices at other locations, the same location, or even othercomponents within the same system. For example, the communication unit808 may include a modem, a network card (wireless or wired), an infraredcommunication device, a wireless communication device (such as anantenna), and/or chipset (such as a Bluetooth device, an 802.6 device(e.g., Metropolitan Area Network (MAN)), a WiFi device, a WiMax device,a cellular communication device, etc.), and/or the like. Thecommunication unit 808 may permit data to be exchanged with a networkand/or any other devices or systems, such as those described in thepresent disclosure.

The operating system 810 may be configured to manage hardware andsoftware resources of the computer system 800 and configured to providecommon services for the computer system 800.

The user interface 812 may include any device configured to allow a userto interface with the computer system 800. For example, the userinterface 812 may include a display, such as an LCD, LED, or otherdisplay, that is configured to present video, text, application userinterfaces, and other data as directed by the processor 802. The userinterface 812 may further include a mouse, a track pad, a keyboard, atouchscreen, volume controls, other buttons, a speaker, a microphone, acamera, any peripheral device, or other input or output device. The userinterface 812 may receive input from a user and provide the input to theprocessor 802. Similarly, the user interface 812 may present output to auser.

The module 812 may be one or more computer-readable instructions storedon one or more non-transitory computer-readable media, such as thememory 804 or the file system 806, that, when executed by the processor802, is configured to perform one or more of the actions of the methods400 and/or 700 described with reference to FIGS. 4 and 7, respectively.In some embodiments, the module 814 may be part of the operating system810 or may be part of an application of the computer system 800, or maybe some combination thereof.

Modifications, additions, or omissions may be made to the computersystem 800 without departing from the scope of the present disclosure.For example, although each is illustrated as a single component in FIG.8, any of the components 802-814 of the computer system 800 may includemultiple similar components that function collectively and arecommunicatively coupled. Further, although illustrated as a singlecomputer system, it is understood that the computer system 800 mayinclude multiple physical or virtual computer systems that are networkedtogether, such as in a cloud computing environment, a multitenancyenvironment, or a virtualization environment.

As indicated above, the embodiments described herein may include the useof a computer (e.g., the processor 130 of FIG. 1) including variouscomputer hardware or software modules, as discussed in greater detailbelow. Further, as indicated above, embodiments described herein may beimplemented using computer-readable media (e.g., the memory 404 or filesystem 140 of FIG. 1) for carrying or having computer-executableinstructions or data structures stored thereon.

As used in the present disclosure, the terms “module” or “component” mayrefer to specific hardware implementations configured to perform theactions of the module or component and/or software objects or softwareroutines that may be stored on and/or executed by hardware (e.g.,computer-readable media, processing devices, etc.) of the computingsystem. In some embodiments, the different components, modules, engines,and services described in the present disclosure may be implemented asobjects or processes that execute on the computing system (e.g., asseparate threads). While some of the system and methods described in thepresent disclosure are generally described as being implemented insoftware (stored on and/or executed by hardware), specific hardwareimplementations or a combination of software and specific hardwareimplementations are also possible and contemplated. In the presentdescription, a “computing entity” may be any computing system aspreviously defined in the present disclosure, or any module orcombination of modulates running on a computing system.

Terms used in the present disclosure and especially in the appendedclaims (e.g., bodies of the appended claims) are generally intended as“open” terms (e.g., the term “including” should be interpreted as“including, but not limited to,” the term “having” should be interpretedas “having at least,” the term “includes” should be interpreted as“includes, but is not limited to,” the term “containing” should beinterpreted as “containing, but not limited to,” etc.).

Additionally, if a specific number of an introduced claim recitation isintended, such an intent will be explicitly recited in the claim, and inthe absence of such recitation no such intent is present. For example,as an aid to understanding, the following appended claims may containusage of the introductory phrases “at least one” and “one or more” tointroduce claim recitations. However, the use of such phrases should notbe construed to imply that the introduction of a claim recitation by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim recitation to embodiments containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should be interpreted to mean “at least one”or “one or more”); the same holds true for the use of definite articlesused to introduce claim recitations.

In addition, even if a specific number of an introduced claim recitationis explicitly recited, those skilled in the art will recognize that suchrecitation should be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, means at least two recitations, or two or more recitations).Furthermore, in those instances where a convention analogous to “atleast one of A, B, and C, etc.” or “one or more of A, B, and C, etc.” isused, in general such a construction is intended to include A alone, Balone, C alone, A and B together, A and C together, B and C together, orA, B, and C together, etc.

Further, any disjunctive word or phrase presenting two or morealternative terms, whether in the description, claims, or drawings,should be understood to contemplate the possibilities of including oneof the terms, either of the terms, or both terms. For example, thephrase “A or B” should be understood to include the possibilities of “A”or “B” or “A and B.”

All examples and conditional language recited in the present disclosureare intended for pedagogical objects to aid the reader in understandingthe disclosure and the concepts contributed by the inventor tofurthering the art, and are to be construed as being without limitationto such specifically recited examples and conditions. Althoughembodiments of the present disclosure have been described in detail,various changes, substitutions, and alterations could be made heretowithout departing from the spirit and scope of the present disclosure.

What is claimed is:
 1. A method, comprising: detecting an object withina field of view of a radar using a radar signal; tracking movement ofthe object through the field of view of the radar; triggering a camerato capture a plurality of images of the object based on the movement ofthe object; detecting the object in the plurality of images; combiningdata of the radar signal with data of the camera to estimate a positionof the object; identifying a radar signal track generated by the motionof the object based on the combined data; estimating a trajectory of theobject based on identifying the radar signal track.
 2. The method ofclaim 1, wherein triggering the camera further comprises: determining apre-determined amount of time has passed since an onset of tracking themovement of the object; and triggering the camera to capture theplurality of images based on a passage of the pre-determined amount oftime.
 3. The method of claim 1, wherein triggering the camera furthercomprises: determining an amplitude of a radar signal associated withtracking the movement of the object; and triggering the camera tocapture the plurality of images based on determining the amplitude ofthe radar signal.
 4. The method of claim 1, wherein triggering thecamera further comprises: determining a pattern of a radar signalassociated with tracking the motion of the object; matching the patternof the radar signal to a previously known pattern stored in memory; andtriggering the camera to capture the plurality of images based onmatching the pattern of the radar signal with the previously knownpattern.
 5. The method of claim 1, wherein triggering the camera furthercomprises: detecting the movement of a second object associated with theobject; and triggering the camera to capture the plurality of imagesbased on detecting the movement of the second object.
 6. The method ofclaim 1, wherein identifying the radar signal track further comprises:generating a Doppler radar signal corresponding to a radial velocity ofthe object.
 7. The method of claim 1, wherein estimating the position ofthe object further comprises: determining an elevation and an azimuthangle of the object in three-dimensional spherical coordinates.
 8. Themethod of claim 7, further comprising: converting the position of theobject from three-dimensional spherical coordinates to Cartesiancoordinates.
 9. The method of claim 1, wherein estimating the trajectoryof the object further comprises: determining an object speed, a verticallaunch angle, a horizontal launch angle, or a combination thereof. 10.The method of claim 1, further comprising: determining a point ofinterest associated with the object, the object having an irregularprofile, wherein triggering the camera further comprises: triggering thecamera and a second camera to capture the plurality of images of theobject.
 11. The method of claim 1, wherein detecting the object furthercomprises: detecting the object located closest to a radar systemassociated with the field of view of the radar.
 12. A system,comprising: a processor; a radar device in electronic communication withthe processor; a camera in electronic communication with the processor;memory in electronic communication with the processor; instructionsstored in the memory, the instructions being executable by the processorto: detect an object within a field of view of a radar using a radarsignal; track movement of the object through the field of view of theradar; trigger a camera to capture a plurality of images of the objectbased on the movement of the object; detect the object in the pluralityof images; combine data of the radar signal with data of the camera toestimate a position of the object; identify a radar signal trackgenerated by the motion of the object based on the combined data;estimate a trajectory of the object based on identifying the radarsignal track.
 13. The system of claim 12, wherein when the processortriggers the camera, the instructions are further executable to:determine a pre-determined amount of time has passed since an onset oftracking the movement of the object; and trigger the camera to capturethe plurality of images based on a passage of the pre-determined amountof time.
 14. The system of claim 12, wherein when the processor triggersthe camera, the instructions are further executable to: determine anamplitude of a radar signal associated with tracking the movement of theobject; and trigger the camera to capture the plurality of images basedon determining the amplitude of the radar signal.
 15. The system ofclaim 12, wherein when the processor triggers the camera, theinstructions are further executable to: determine a pattern of a radarsignal associated with tracking the movement of the object; match thepattern of the radar signal to a previously known pattern stored inmemory; and trigger the camera to capture the plurality of images basedon matching the pattern of the radar signal with the previously knownpattern.
 16. The system of claim 12, wherein when the processor triggersthe camera, the instructions are further executable to: detect themovement of a second object associated with the object; and trigger thecamera to capture the plurality of images based on detecting themovement of the second object.
 17. The system of claim 12, wherein whenthe processor identifies the radar signal track, the instructions arefurther executable to: generate a Doppler radar signal corresponding toa radial velocity of the object.
 18. The system of claim 12, whereinwhen the processor estimates the position of the object, theinstructions are further executable to: determine an elevation and anazimuth angle of the object in three-dimensional spherical coordinates.19. The system of claim 18, wherein the instructions are furtherexecutable by the processor to cause the system to: convert the positionof the object from three-dimensional spherical coordinates to Cartesiancoordinates.
 20. A non-transitory computer-readable medium storingcomputer-executable code, the code executable by a processor to cause asystem to: detect an object within a field of view of a radar using aradar signal; track movement of the object through the field of view ofthe radar; trigger a camera to capture a plurality of images of theobject based on the movement of the object; detect the object in theplurality of images; combine data of the radar signal with data of thecamera to estimate a position of the object; identify a radar signaltrack generated by the motion of the object based on the combined data;estimate a trajectory of the object based on identifying the radarsignal track.